Operationalizing Programmatic Assessment: The CBME Programmatic Assessment Practice Guidelines.

PROBLEM: Assessing the development and achievement of competence requires multiple formative and summative assessment strategies and the coordinated efforts of trainees and faculty (who often serve in multiple roles, such as academic advisors, program directors, and competency committee members). Operationalizing programmatic assessment (PA) in competency-based medical education (CBME) requires comprehensive practice guidelines, written in accessible language with descriptions of stakeholder activities, to move assessment theory into practice and to help guide the trainees and faculty who enact PA. APPROACH: Informed by the Appraisal of Guidelines for Research and Evaluation II (AGREE II) framework, the authors used a multiphase, multimethod approach to develop the CBME Programmatic Assessment Practice Guidelines (PA Guidelines). The 9 guidelines are organized by phases of assessment and include descriptions of stakeholder activities. A user guide provides a glossary of key terms and summarizes how the guidelines can be used by different stakeholder groups across postgraduate medical education (PGME) contexts. The 4 phases of guideline development, including internal stakeholder consultations and external expert review, occurred between August 2016 and March 2020. OUTCOMES: Local stakeholders and external experts agreed that the PA Guidelines hold potential for guiding initial operationalization and ongoing refinement of PA in CBME by individual stakeholders, residency programs, and PGME institutions. Since July 2020, the PA Guidelines have been used at Queen's University to inform faculty and resident development initiatives, including online CBME modules for faculty, workshops for academic advisors/competence committee members, and a guide that supports incoming residents' transition to CBME. NEXT STEPS: Research exploring the use of the PA Guidelines and user guide in multiple programs and institutions will gather further evidence of their acceptability and utility for guiding operationalization of PA in different contexts.

Translating outcome frameworks to assessment programmes: Implications for validity.

OBJECTIVES: Competency-based medical education (CBME) requires that educators structure assessment of clinical competence using outcome frameworks. Although these frameworks may serve some outcomes well (e.g. represent eventual practice), translating these into workplace-based assessment plans may undermine validity and, therefore, trustworthiness of assessment decisions due to a number of competing factors that may not always be visible or their impact knowable. Explored here is the translation process from outcome framework to formative and summative assessment plans in postgraduate medical education (PGME) in three Canadian universities. METHODS: We conducted a qualitative study involving in-depth semi-structured interviews with leaders of PGME programmes involved in assessment and/or CBME implementation, with a focus on their assessment-based translational activities and evaluation strategies. Interviews were informed by Callon's theory of translation. Our analytical strategy involved directed content analysis, allowing us to be guided by Kane's validity framework, whilst still participating in open coding and analytical memo taking. We then engaged in axial coding to systematically explore themes across the dataset, various situations and our conceptual framework. RESULTS: Twenty-four interviews were conducted involving 15 specialties across three universities. Our results suggest: (i) using outcomes frameworks for assessment is necessary for good assessment but are also viewed as incomplete constructs; (ii) there are a number of social and practical negotiations with competing factors that displace validity as a core influencer in assessment planning, including implementation, accreditation and technology; and (iii) validity exists as threatened, uncertain and assumed due to a number of unchecked assumptions and reliance on surrogates. CONCLUSIONS: Translational processes in CBME involve negotiating with numerous influencing actors and institutions that, from an assessment perspective, provide challenges for assessment scientists, institutions and educators to contend with. These processes are challenging validity as a core element of assessment designs. Educators must reconcile these influences when preparing for or structuring validity arguments.

Developing and successfully implementing a competency-based portfolio assessment system in a postgraduate family medicine residency program.

The use of portfolios in postgraduate medical residency education to support competency development is increasing; however, the processes by which these assessment systems are designed, implemented, and maintained are emergent. The authors describe the needs assessment, development, implementation, and continuing quality improvement processes that have shaped the Portfolio Assessment Support System (PASS) used by the postgraduate family medicine program at Queen's University since 2009. Their description includes the impetus for change and contextual realities that guided the effort, plus the processes used for selecting assessment components and developing strategic supports. The authors discuss the identification of impact measures at the individual, programmatic, and institutional levels and the ways the department uses these to monitor how PASS supports competency development, scaffolds residents' self-regulated learning skills, and promotes professional identity formation. They describe the "academic advisor" role and provide an appendix covering the portfolio elements. Reflection elements include learning plans, clinical question logs, confidence surveys, and reflections about continuity of care and significant incidents. Learning module elements cover the required, online bioethics, global health, and consult-request modules. Assessment elements cover each resident's research project, clinical audits, presentations, objective structured clinical exam and simulated office oral exam results, field notes, entrustable professional activities, multisource feedback, and in-training evaluation reports. Document elements are the resident's continuing medical education activities including procedures log, attendance log, and patient demographic summaries.The authors wish to support others who are engaged in the systematic portfolio-design process or who may adapt aspects of PASS for their local programs.